Test Enumeration
The book has now been published and the content of this chapter has likely changed substanstially.Please see page 399 of xUnit Test Patterns for the latest information.
Also known as: Test Suite Factory
How does the Test Runner know what tests to run?
The test automater manually writes the code that enumerates all tests that belong to the test suite.
Sketch Test Enumeration embedded from Test Enumeration.gif
Given that we have written a number of Test Methods (page X) on one or more Testcase Classes (page X), we need to give the Test Runner (page X) some way to find the tests.
Test Enumeration is the way we do it when we do not have support for Test Discovery (page X).
How It Works
The test automater manually writes the code that enumerates all the Test Methods that belong to the test suite and/or all the Test Suite Objects (page X) that belong to a Suite of Suites (see Test Suite Object). This is typically done by implementing the method suite on either a Testcase Class for Test Method Enumeration or on a Test Suite Factory for Test Suite Enumeration.
When To Use It
We need to use Test Enumeration if our Test Automation Framework (page X) does not support Test Discovery. We can also choose to use Test Enumeration when we wish to define a Named Test Suite (page X) that consists of a subset of tests (A Smoke Test[SCM] suite is a good example of this.) chosen from other test suites and the framework does not support Test Selection (page X).
Many members of the xUnit family support Test Discovery at the Test Method level but force us to use Test Enumeration at the Testcase Class level.
Implementation Notes
Building the Suite of Suites to be executed by the Test Runner involves two things. First, we must find all the Test Methods to be included in each Test Suite Object and second, we must find all the Test Suite Objects to be included in the test run though not necessarily in this order. Each of these steps may be done manually via Test Method Enumeration and Test Suite Enumeration or automatically via Test Method Discovery (see Test Discovery) and Testcase Class Discovery (see Test Discovery). When done manually, we typically use a "Test Suite Factory" that returns the Test Suite Object.
Variation: Test Suite Enumeration
Many members of the xUnit family require that we provide a Test Suite Factory that builds the top-level Suite of Suites (often called "AllTests") as means for us to specify what Test Suite Objects we would like to include in a test run. We do this by providing a class method on a factory class; this Factory Method[GOF] is called suite in most members of the xUnit family. Inside the suite method we use calls to methods such as addTest to add each nested Test Suite Object to the suite we are building.
This approach is fairly flexible but it can result in Lost Tests (see Production Bugs on page X). The alternative is to let the development tools build the AllTests Suite (see Named Test Suite) automatically or to use a Test Runner that finds all the test suites in a file system directory automatically. For example, NUnit provides a built in mechanism that implements Testcase Class Discovery at the assembly level. Third party tools such as Ant can also be used to find all the tests in a directory structure.
Even in statically-typed languages like Java, the Test Suite Factory (see Test Enumeration on page X) does not need to subclass a specific class or implement a specific interface because the only dependency is on the generic Test Suite Object class it returns and the Testcase Classes or Test Suite Factories it asks for the nested suites.
Variation: Test Method Enumeration
Many members of the xUnit family now support Test Method Discovery but if we are using one that does not, we need to find all the Test Methods in a Testcase Class, turn them into Testcase Objects (page X) and put them into a Test Suite Object. We implement Test Method Enumeration by providing a class method, typically called suite on the Testcase Class itself.
The capability to construct an object that calls an arbitrary method is often inherited from the Test Automation Framework via a Testcase Superclass (page X) or mixed in via a class attribute or Include directive. In some members of the xUnit family, this Pluggable Behavior[SBPP] capability is provided by a separate class (see the CppUnit example below.)
Variation: Direct Test Method Invocation
In the pure procedural world where we cannot treat a Test Method as an object or data item, we have no choice but to hand-code a Test Suite Procedure (see Test Suite Object) for each test suite. The procedure calls each Test Method (or other Test Suite Procedures) one by one.
Example: Test Method Enumeration in CppUnit
Early versions of most xUnit family members required that the test automater add each Test Method manually. Those that cannot use reflection still require it. Here is an example from an older version of CppUnit that uses this approach:
public:
static CppUnit::Test *suite()
{
CppUnit::TestSuite *suite = new CppUnit::TestSuite( "ComplexNumberTest" );
suite>addTest( new CppUnit::TestCaller<ComplexNumberTest>( "testEquality",
&ComplexNumberTest::testEquality ) );
suite>addTest( new CppUnit::TestCaller<ComplexNumberTest>( "testAddition",
&ComplexNumberTest::testAddition ) );
return suite;
}
Example ManualTestMethodEnumeration embedded from CPP/CppUnitTestSuitConstruction.cpp
This example also illustrates how CppUnit wraps each Test Method with an instance of a class (TestCaller) to turn it into a Testcase Object.
Example: Test Method Invocation (hard-coded)
The following example is from a test suite for a program written in VBA (Visual Basic for Applications, the macro language used in Microsoft Office products) which lacks support for objects:
Sub TestAllStoryMacros() Call TestActivitySorting
Call TestStoryHiding
Call ReportSuccess("All Story Macros")
End Sub
Example ManualTestMethodInvokation embedded from VBA/ClearTrackTest.vba
Example: Test Suite Enumeration
We can use Test Suite Enumeration when the Test Automation Framework does not support Test Discovery or when we want to define a Named Test Suite that includes only a subset of the tests.
The main drawback of using Test Suite Enumeration for running all the tests is the possibility of Lost Tests caused by forgetting to include a new test suite in the AllTests Suite. This can be reduced by paying attention to the number of tests that were run when we first checked out the code and ensuring that the number run just before check-in goes up by the number of new tests we added.
public class AllTests {
public static Test suite() {
TestSuite suite = new TestSuite("Test for allJunitTests");
//$JUnit-BEGIN$
suite.addTestSuite( com.clrstream.camug.example.test.InvoiceTest.class);
suite.addTest(com.clrstream.ex7.test.AllTests.suite());
suite.addTest(com.clrstream.ex8.test.AllTests.suite());
suite.addTestSuite( com.xunitpatterns.guardassertion.Example.class);
//$JUnit-END$
return suite;
}
}
Example AllTestsGenerated embedded from java/allJunitTests/AllTests.java
In this example, we are taking advantage of the IDE's ability to (re)generate the AllTests suite for us. (Eclipse will regenerate all the code between the two marker comments whenever we request it.) We still need to remember to regenerate the suite occasionally but this goes a long way to avoiding Lost Tests in the absence of Test Discovery.
Copyright © 2003-2008 Gerard Meszaros all rights reserved